{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Logging setup\n",
    "import logging\n",
    "\n",
    "logging.basicConfig(level=logging.INFO)\n",
    "logging.getLogger().setLevel(level=logging.ERROR)\n",
    "logging.getLogger('orion').setLevel(level=logging.INFO)\n",
    "\n",
    "import warnings\n",
    "warnings.simplefilter(\"ignore\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Orion Tutorial\n",
    "\n",
    "In the following steps we will learn how to set Orion up, run pipelines to detect anomalies\n",
    "on our timeseries and then explore the results.\n",
    "\n",
    "Overall, the steps that we will perform are:\n",
    "\n",
    "1. Add _Datasets_, _Signals_, _Templates_, _Pipelines_ and _Experiments_ to our Database.\n",
    "2. Create start _Dataruns_, which create _Signalruns_ and _Events_.\n",
    "3. Explore the _Signalrun_ results and the detected _Events_.\n",
    "4. Add _Annotations_ to the existing _Events_ as well as new manual _Events_.\n",
    "\n",
    "## Creating an instance of the OrionDBExplorer\n",
    "\n",
    "In order to connect to the database, all you need to do is import and create an instance of the\n",
    "`OrionDBExplorer` class.\n",
    "\n",
    "Note that, because of the dynamic schema-less nature of MongoDB, no database initialization\n",
    "or table creation is needed. All you need to do start using a new database is create the\n",
    "`OrionDBExplorer` instance with the right connection details and start using it!\n",
    "\n",
    "In order to create the `OrionDBExplorer` instance you will need to pass:\n",
    "\n",
    "* `user`: An identifier of the user that is running Orion.\n",
    "* `database`: The name of the MongoDB database to use. This is optional and defaults to `orion`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from orion.db import OrionDBExplorer\n",
    "\n",
    "orex = OrionDBExplorer(user='my_username', database='orion-usage-example')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This will directly create a connection to the database named `'orion'` at the default\n",
    "MongoDB host, `localhost`, and port, `27017`.\n",
    "\n",
    "In case you wanted to connect to a different database, host or port, or in case user authentication\n",
    "is enabled in your MongoDB instance, you can pass a dictionary or a path to a JSON file containing\n",
    "any required additional arguments:\n",
    "\n",
    "* `host`: Hostname or IP address of the MongoDB Instance. Defaults to `'localhost'`.\n",
    "* `port`: Port to which MongoDB is listening. Defaults to `27017`.\n",
    "* `username`: username to authenticate with.\n",
    "* `password`: password to authenticate with.\n",
    "* `authentication_source`: database to authenticate against."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once we have created the `OrionDBExplorer` instance, and to be sure that we are ready to follow\n",
    "the tutorial, let's do the following two set-up setps:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Drop the `orion-usage-example` database\n",
    "\n",
    "**WARNING**: This will remove all the data that exists in this database!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "orex.drop_database()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2. Make sure to have downloaded some demo data using the `orion.data.download_demo()` function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:orion.data:Downloading Orion Demo Data to folder orion-data\n"
     ]
    }
   ],
   "source": [
    "from orion.data import download_demo\n",
    "\n",
    "download_demo()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This will create a folder called `orion-data` in your current directory with the 3 CSV files\n",
    "that we will use later on."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setting up the Orion Environment\n",
    "\n",
    "The first thing that you will need to do to start using **Orion** with a Database will be\n",
    "to add information about your data and your pipelines.\n",
    "\n",
    "This can be done by using the methods of the `OrionDBExplorer` class that are documenteted below,\n",
    "which allow creating the corresponding objects in the Database.\n",
    "\n",
    "### Add a Dataset\n",
    "\n",
    "In order to add a dataset you can use the `add_dataset` method, which has the following arguments:\n",
    "\n",
    "* `name (str)`: Name of the dataset\n",
    "* `entity (str)`: Name or Id of the entity which this dataset is associated to\n",
    "\n",
    "Let's create the `Demo Dataset` that we will use for our demo."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset = orex.add_dataset(\n",
    "    name='Demo Dataset',\n",
    "    entity='Orion',\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This call will try to create a new _Dataset_ object in the database and return it.\n",
    "\n",
    "We can now see the _Dataset_ that we just created using the `get_datasets` method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>dataset_id</th>\n",
       "      <th>created_by</th>\n",
       "      <th>entity</th>\n",
       "      <th>insert_time</th>\n",
       "      <th>name</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>602c91be219a13a319ca725c</td>\n",
       "      <td>my_username</td>\n",
       "      <td>Orion</td>\n",
       "      <td>2021-02-17 03:47:10.234</td>\n",
       "      <td>Demo Dataset</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 dataset_id   created_by entity             insert_time  \\\n",
       "0  602c91be219a13a319ca725c  my_username  Orion 2021-02-17 03:47:10.234   \n",
       "\n",
       "           name  \n",
       "0  Demo Dataset  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "orex.get_datasets()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Add a Signal\n",
    "\n",
    "The next step is to add Signals. This can be done with the `add_signal` method, which expects:\n",
    "\n",
    "* `name (str)`: Name of the signal\n",
    "* `dataset (Dataset or ObjectID)`: Dataset Object or Dataset Id.\n",
    "* `start_time (int)`: (Optional) minimum timestamp to be used for this signal. If not given, it\n",
    "  defaults to the minimum timestamp found in the data.\n",
    "* `stop_time (int)`: (Optional) maximum timestamp to be used for this signal. If not given, it\n",
    "  defaults to the maximum timestamp found in the data.\n",
    "* `data_location (str)`: URI of the dataset\n",
    "* `timestamp_column (int)`: (Optional) index of the timestamp column. Defaults to 0.\n",
    "* `value_column (int)`: (Optional) index of the value column. Defaults to 1.\n",
    "\n",
    "For example, adding the `S-1` signal to the Demo Dataset that we just created could be done like\n",
    "this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Signal: Signal object>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "orex.add_signal(\n",
    "    name='S-1',\n",
    "    dataset=dataset,\n",
    "    data_location='orion-data/S-1.csv'\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Additionally, we can also add all the signals that exist inside a folder by using the `add_signals`\n",
    "method, passing a `signals_path`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "orex.add_signals(\n",
    "    dataset=dataset,\n",
    "    signals_path='orion-data'\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After this is done, we can see that one signal has been created for each one of the CSV\n",
    "files that we downloaded before."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>signal_id</th>\n",
       "      <th>created_by</th>\n",
       "      <th>data_location</th>\n",
       "      <th>dataset</th>\n",
       "      <th>insert_time</th>\n",
       "      <th>name</th>\n",
       "      <th>start_time</th>\n",
       "      <th>stop_time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>602c91be219a13a319ca725d</td>\n",
       "      <td>my_username</td>\n",
       "      <td>orion-data/S-1.csv</td>\n",
       "      <td>602c91be219a13a319ca725c</td>\n",
       "      <td>2021-02-17 03:47:10.627</td>\n",
       "      <td>S-1</td>\n",
       "      <td>1222819200</td>\n",
       "      <td>1442016000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>602c91be219a13a319ca725f</td>\n",
       "      <td>my_username</td>\n",
       "      <td>orion-data/P-1.csv</td>\n",
       "      <td>602c91be219a13a319ca725c</td>\n",
       "      <td>2021-02-17 03:47:10.920</td>\n",
       "      <td>P-1</td>\n",
       "      <td>1222819200</td>\n",
       "      <td>1468540800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>602c91be219a13a319ca7260</td>\n",
       "      <td>my_username</td>\n",
       "      <td>orion-data/E-1.csv</td>\n",
       "      <td>602c91be219a13a319ca725c</td>\n",
       "      <td>2021-02-17 03:47:10.931</td>\n",
       "      <td>E-1</td>\n",
       "      <td>1222819200</td>\n",
       "      <td>1468951200</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  signal_id   created_by       data_location  \\\n",
       "0  602c91be219a13a319ca725d  my_username  orion-data/S-1.csv   \n",
       "1  602c91be219a13a319ca725f  my_username  orion-data/P-1.csv   \n",
       "2  602c91be219a13a319ca7260  my_username  orion-data/E-1.csv   \n",
       "\n",
       "                    dataset             insert_time name  start_time  \\\n",
       "0  602c91be219a13a319ca725c 2021-02-17 03:47:10.627  S-1  1222819200   \n",
       "1  602c91be219a13a319ca725c 2021-02-17 03:47:10.920  P-1  1222819200   \n",
       "2  602c91be219a13a319ca725c 2021-02-17 03:47:10.931  E-1  1222819200   \n",
       "\n",
       "    stop_time  \n",
       "0  1442016000  \n",
       "1  1468540800  \n",
       "2  1468951200  "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "orex.get_signals(dataset=dataset)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Add a Template\n",
    "\n",
    "The next thing we need to add is a _Template_ to the Database using the `add_template` method.\n",
    "\n",
    "This method expects:\n",
    "\n",
    "* `name (str)`: Name of the template.\n",
    "* `template (dict or str)`: Optional. Specification of the template to use, which can be one of:\n",
    "    * An MLPipeline instance\n",
    "    * The name of a registered template\n",
    "    * a dict containing the MLPipeline details\n",
    "    * The path to a pipeline JSON file.\n",
    "    \n",
    "**Orion** comes with a few templates ready to be used, so let's have a look at the ones that exist\n",
    "using the `orion.get_available_templates` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['azure', 'arima', 'tadgan', 'lstm_dynamic_threshold', 'dummy']"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from orion.analysis import get_available_templates\n",
    "\n",
    "get_available_templates()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now let's create a _Template_ using the `lstm_dynamic_threshold` template."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "template = orex.add_template(\n",
    "    name='lstmdt',\n",
    "    template='lstm_dynamic_threshold',\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now see the _Template_ that we just created"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>template_id</th>\n",
       "      <th>created_by</th>\n",
       "      <th>insert_time</th>\n",
       "      <th>name</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>6026e925b402c68ae42b1105</td>\n",
       "      <td>my_username</td>\n",
       "      <td>2021-02-12 20:46:29.187</td>\n",
       "      <td>lstmdt</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                template_id   created_by             insert_time    name\n",
       "0  6026e925b402c68ae42b1105  my_username 2021-02-12 20:46:29.187  lstmdt"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "orex.get_templates()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Also, during this step, apart from a _Template_ object, a _Pipeline_ object has also been\n",
    "registred with the same name as the _Template_ and using the default hyperparameter values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>pipeline_id</th>\n",
       "      <th>created_by</th>\n",
       "      <th>insert_time</th>\n",
       "      <th>name</th>\n",
       "      <th>template</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>6026e925b402c68ae42b1106</td>\n",
       "      <td>my_username</td>\n",
       "      <td>2021-02-12 20:46:29.416</td>\n",
       "      <td>lstmdt</td>\n",
       "      <td>6026e925b402c68ae42b1105</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                pipeline_id   created_by             insert_time    name  \\\n",
       "0  6026e925b402c68ae42b1106  my_username 2021-02-12 20:46:29.416  lstmdt   \n",
       "\n",
       "                   template  \n",
       "0  6026e925b402c68ae42b1105  "
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "orex.get_pipelines()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, if we want to use a configuration different from the default, we might want to\n",
    "create another _Pipeline_ with custom hyperparameter values.\n",
    "\n",
    "In order to do this we will need to call the `add_pipeline` method passing:\n",
    "\n",
    "* `name (str)`: Name given to this pipeline\n",
    "* `template (Template or ObjectID)`: Template or the corresponding id.\n",
    "* `hyperparameters (dict or str)`: dict containing the hyperparameter details or path to the\n",
    "  corresponding JSON file. Optional.\n",
    "\n",
    "For example, if we want to specify a different number of epochs for the LSTM primitive of the\n",
    "pipeline that we just created we will run:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "new_hyperparameters = {\n",
    "   'keras.Sequential.LSTMTimeSeriesRegressor#1': {\n",
    "       'epochs': 1,\n",
    "       'verbose': True\n",
    "   }\n",
    "}\n",
    "pipeline = orex.add_pipeline(\n",
    "   name='lstmdt_1_epoch',\n",
    "   template=template,\n",
    "   hyperparameters=new_hyperparameters,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And we can see how a new _Pipeline_ was created in the Database."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>pipeline_id</th>\n",
       "      <th>created_by</th>\n",
       "      <th>insert_time</th>\n",
       "      <th>name</th>\n",
       "      <th>template</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>6026e925b402c68ae42b1106</td>\n",
       "      <td>my_username</td>\n",
       "      <td>2021-02-12 20:46:29.416</td>\n",
       "      <td>lstmdt</td>\n",
       "      <td>6026e925b402c68ae42b1105</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>6026e925b402c68ae42b1107</td>\n",
       "      <td>my_username</td>\n",
       "      <td>2021-02-12 20:46:29.713</td>\n",
       "      <td>lstmdt_1_epoch</td>\n",
       "      <td>6026e925b402c68ae42b1105</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                pipeline_id   created_by             insert_time  \\\n",
       "0  6026e925b402c68ae42b1106  my_username 2021-02-12 20:46:29.416   \n",
       "1  6026e925b402c68ae42b1107  my_username 2021-02-12 20:46:29.713   \n",
       "\n",
       "             name                  template  \n",
       "0          lstmdt  6026e925b402c68ae42b1105  \n",
       "1  lstmdt_1_epoch  6026e925b402c68ae42b1105  "
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "orex.get_pipelines()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Add an Experiment\n",
    "\n",
    "Once we have a _Dataset_ with _Signals_ and a _Template_, we are ready to add an\n",
    "_Experiment_.\n",
    "\n",
    "In order to run an _Experiment_ we will need to:\n",
    "\n",
    "1. Get the _Dataset_ and the list of _Signals_ that we want to run the _Experiment_ on.\n",
    "2. Get the _Template_ which we want to use for the _Experiment_\n",
    "3. Call the `add_experiment` method passing all these with an experiment, a project name and a\n",
    "   username.\n",
    "\n",
    "For example, if we want to create an experiment using the _Dataset_, the _Signals_ and the\n",
    "_Template_ that we just created, we will use:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "experiment = orex.add_experiment(\n",
    "    name='My Experiment',\n",
    "    project='My Project',\n",
    "    template=template,\n",
    "    dataset=dataset,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This will create an _Experiment_ object in the database using the indicated _Template_\n",
    "and all the _Signals_ from the given _Dataset_."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>experiment_id</th>\n",
       "      <th>created_by</th>\n",
       "      <th>dataset</th>\n",
       "      <th>insert_time</th>\n",
       "      <th>name</th>\n",
       "      <th>project</th>\n",
       "      <th>signals</th>\n",
       "      <th>template</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>6026e925b402c68ae42b1108</td>\n",
       "      <td>my_username</td>\n",
       "      <td>6026e920b402c68ae42b1100</td>\n",
       "      <td>2021-02-12 20:46:29.747</td>\n",
       "      <td>My Experiment</td>\n",
       "      <td>My Project</td>\n",
       "      <td>[6026e921b402c68ae42b1101, 6026e921b402c68ae42...</td>\n",
       "      <td>6026e925b402c68ae42b1105</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              experiment_id   created_by                   dataset  \\\n",
       "0  6026e925b402c68ae42b1108  my_username  6026e920b402c68ae42b1100   \n",
       "\n",
       "              insert_time           name     project  \\\n",
       "0 2021-02-12 20:46:29.747  My Experiment  My Project   \n",
       "\n",
       "                                             signals                  template  \n",
       "0  [6026e921b402c68ae42b1101, 6026e921b402c68ae42...  6026e925b402c68ae42b1105  "
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "orex.get_experiments()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Starting a Datarun\n",
    "\n",
    "Once we have created our _Experiment_ object we are ready to start executing _Pipelines_ on our\n",
    "_Signals_.\n",
    "\n",
    "For this we will need to use the `orion.runner.start_datarun` function, which expects:\n",
    "\n",
    "* `orex (OrionExplorer)`: The `OrionDBExplorer` instance.\n",
    "* `experiment (Experiment or ObjectID)`: Experiment object or the corresponding ID.\n",
    "* `pipeline (Pipeline or ObjectID)`: Pipeline object or the corresponding ID.\n",
    "\n",
    "This will create a _Datarun_ object for this _Experiment_ and _Pipeline_ in the database,\n",
    "and then it will start creating and executing _Signalruns_, one for each _Signal_ in the _Experiment_."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's trigger a _Datarun_ using the `lstmdt_1_epoch` _Pipeline_ that we created."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:orion.runner:Datarun 6026e926b402c68ae42b1109 started\n",
      "INFO:orion.runner:Signalrun 6026e926b402c68ae42b110a started\n",
      "INFO:orion.runner:Running pipeline lstmdt_1_epoch on signal S-1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 7919 samples, validate on 1980 samples\n",
      "Epoch 1/1\n",
      "7919/7919 [==============================] - 37s 5ms/step - loss: 0.1948 - mse: 0.1948 - val_loss: 0.2965 - val_mse: 0.2965\n",
      "9899/9899 [==============================] - 13s 1ms/step\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:orion.runner:Processing pipeline lstmdt_1_epoch predictions on signal S-1\n",
      "INFO:orion.runner:Signalrun 6026e9a4b402c68ae42b110d started\n",
      "INFO:orion.runner:Running pipeline lstmdt_1_epoch on signal P-1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 8901 samples, validate on 2226 samples\n",
      "Epoch 1/1\n",
      "8901/8901 [==============================] - 41s 5ms/step - loss: 0.0965 - mse: 0.0965 - val_loss: 0.0535 - val_mse: 0.0535\n",
      "11127/11127 [==============================] - 15s 1ms/step\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:orion.runner:Processing pipeline lstmdt_1_epoch predictions on signal P-1\n",
      "INFO:orion.runner:Signalrun 6026ea39b402c68ae42b110e started\n",
      "INFO:orion.runner:Running pipeline lstmdt_1_epoch on signal E-1\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 8916 samples, validate on 2230 samples\n",
      "Epoch 1/1\n",
      "8916/8916 [==============================] - 47s 5ms/step - loss: 0.2199 - mse: 0.2199 - val_loss: 0.1407 - val_mse: 0.1407\n",
      "11146/11146 [==============================] - 15s 1ms/step\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:orion.runner:Processing pipeline lstmdt_1_epoch predictions on signal E-1\n"
     ]
    }
   ],
   "source": [
    "from orion.runner import start_datarun\n",
    "\n",
    "start_datarun(orex, experiment, pipeline)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Explore the results\n",
    "\n",
    "Once a _Datarun_ has finished, we can see can see its status by using the `orex.get_dataruns` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>datarun_id</th>\n",
       "      <th>end_time</th>\n",
       "      <th>experiment</th>\n",
       "      <th>insert_time</th>\n",
       "      <th>num_events</th>\n",
       "      <th>pipeline</th>\n",
       "      <th>start_time</th>\n",
       "      <th>status</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>6026e926b402c68ae42b1109</td>\n",
       "      <td>2021-02-12 20:54:15.818</td>\n",
       "      <td>6026e925b402c68ae42b1108</td>\n",
       "      <td>2021-02-12 20:46:29.978</td>\n",
       "      <td>3</td>\n",
       "      <td>6026e925b402c68ae42b1107</td>\n",
       "      <td>2021-02-12 20:46:30.082</td>\n",
       "      <td>SUCCESS</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 datarun_id                end_time                experiment  \\\n",
       "0  6026e926b402c68ae42b1109 2021-02-12 20:54:15.818  6026e925b402c68ae42b1108   \n",
       "\n",
       "              insert_time  num_events                  pipeline  \\\n",
       "0 2021-02-12 20:46:29.978           3  6026e925b402c68ae42b1107   \n",
       "\n",
       "               start_time   status  \n",
       "0 2021-02-12 20:46:30.082  SUCCESS  "
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "orex.get_dataruns()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As well as the _Signalruns_ and _Events_ that were created."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>signalrun_id</th>\n",
       "      <th>datarun</th>\n",
       "      <th>end_time</th>\n",
       "      <th>insert_time</th>\n",
       "      <th>num_events</th>\n",
       "      <th>signal</th>\n",
       "      <th>start_time</th>\n",
       "      <th>status</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>6026e926b402c68ae42b110a</td>\n",
       "      <td>6026e926b402c68ae42b1109</td>\n",
       "      <td>2021-02-12 20:48:36.182</td>\n",
       "      <td>2021-02-12 20:46:30.091</td>\n",
       "      <td>2</td>\n",
       "      <td>6026e921b402c68ae42b1101</td>\n",
       "      <td>2021-02-12 20:46:30.187</td>\n",
       "      <td>SUCCESS</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>6026e9a4b402c68ae42b110d</td>\n",
       "      <td>6026e926b402c68ae42b1109</td>\n",
       "      <td>2021-02-12 20:51:05.253</td>\n",
       "      <td>2021-02-12 20:48:36.183</td>\n",
       "      <td>0</td>\n",
       "      <td>6026e921b402c68ae42b1103</td>\n",
       "      <td>2021-02-12 20:48:36.184</td>\n",
       "      <td>SUCCESS</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>6026ea39b402c68ae42b110e</td>\n",
       "      <td>6026e926b402c68ae42b1109</td>\n",
       "      <td>2021-02-12 20:54:15.817</td>\n",
       "      <td>2021-02-12 20:51:05.256</td>\n",
       "      <td>1</td>\n",
       "      <td>6026e921b402c68ae42b1104</td>\n",
       "      <td>2021-02-12 20:51:05.258</td>\n",
       "      <td>SUCCESS</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               signalrun_id                   datarun                end_time  \\\n",
       "0  6026e926b402c68ae42b110a  6026e926b402c68ae42b1109 2021-02-12 20:48:36.182   \n",
       "1  6026e9a4b402c68ae42b110d  6026e926b402c68ae42b1109 2021-02-12 20:51:05.253   \n",
       "2  6026ea39b402c68ae42b110e  6026e926b402c68ae42b1109 2021-02-12 20:54:15.817   \n",
       "\n",
       "              insert_time  num_events                    signal  \\\n",
       "0 2021-02-12 20:46:30.091           2  6026e921b402c68ae42b1101   \n",
       "1 2021-02-12 20:48:36.183           0  6026e921b402c68ae42b1103   \n",
       "2 2021-02-12 20:51:05.256           1  6026e921b402c68ae42b1104   \n",
       "\n",
       "               start_time   status  \n",
       "0 2021-02-12 20:46:30.187  SUCCESS  \n",
       "1 2021-02-12 20:48:36.184  SUCCESS  \n",
       "2 2021-02-12 20:51:05.258  SUCCESS  "
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "datarun = orex.get_datarun(experiment=experiment)\n",
    "signalruns = orex.get_signalruns(datarun=datarun)\n",
    "signalruns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>event_id</th>\n",
       "      <th>insert_time</th>\n",
       "      <th>num_annotations</th>\n",
       "      <th>severity</th>\n",
       "      <th>signal</th>\n",
       "      <th>signalrun</th>\n",
       "      <th>source</th>\n",
       "      <th>start_time</th>\n",
       "      <th>stop_time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>6026eaf7b402c68ae42b110f</td>\n",
       "      <td>2021-02-12 20:54:15.815</td>\n",
       "      <td>0</td>\n",
       "      <td>0.080653</td>\n",
       "      <td>6026e921b402c68ae42b1104</td>\n",
       "      <td>6026ea39b402c68ae42b110e</td>\n",
       "      <td>ORION</td>\n",
       "      <td>1392746400</td>\n",
       "      <td>1395057600</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   event_id             insert_time  num_annotations  \\\n",
       "0  6026eaf7b402c68ae42b110f 2021-02-12 20:54:15.815                0   \n",
       "\n",
       "   severity                    signal                 signalrun source  \\\n",
       "0  0.080653  6026e921b402c68ae42b1104  6026ea39b402c68ae42b110e  ORION   \n",
       "\n",
       "   start_time   stop_time  \n",
       "0  1392746400  1395057600  "
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "signalrun_id = signalruns['signalrun_id'].iloc[-1]\n",
    "orex.get_events(signalrun=signalrun_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Add Manual Events and Annotations\n",
    "\n",
    "If we want to add new events manually, we can do so by calling the `add_event` method and\n",
    "passing:\n",
    "\n",
    "* `start_time (int)`: The timestamp at which the event starts\n",
    "* `stop_time (int)`: The timestamp at which the event ends\n",
    "* `source (str)`: If manual, the string `MANUALLY_CREATED`.\n",
    "* `signal (Signal or ObjectID or str)`: The id to which the Event is associated."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "signal = orex.get_signal(name='P-1')\n",
    "event = orex.add_event(\n",
    "    start_time=1393758300,\n",
    "    stop_time=1408270800,\n",
    "    source='MANUALLY_CREATED',\n",
    "    signal=signal\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And, optionally, add annotations to any of the events."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Annotation: Annotation object>"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "orex.add_annotation(\n",
    "    event=event,\n",
    "    tag='maneuver',\n",
    "    comment='satellite was maneuvering during this period'\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Annotation: Annotation object>"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "unknown_event = orex.get_events().event_id.iloc[0]\n",
    "orex.add_annotation(\n",
    "    event=unknown_event,\n",
    "    tag='unknown',\n",
    "    comment='this needs to be investigated'\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can then see the annotations that we just created"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>annotation_id</th>\n",
       "      <th>comment</th>\n",
       "      <th>event</th>\n",
       "      <th>insert_time</th>\n",
       "      <th>tag</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>6026eaf8b402c68ae42b1111</td>\n",
       "      <td>satellite was maneuvering during this period</td>\n",
       "      <td>6026eaf7b402c68ae42b1110</td>\n",
       "      <td>2021-02-12 20:54:15.970</td>\n",
       "      <td>maneuver</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>6026eaf8b402c68ae42b1112</td>\n",
       "      <td>this needs to be investigated</td>\n",
       "      <td>6026e9a4b402c68ae42b110b</td>\n",
       "      <td>2021-02-12 20:54:16.131</td>\n",
       "      <td>unknown</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              annotation_id                                       comment  \\\n",
       "0  6026eaf8b402c68ae42b1111  satellite was maneuvering during this period   \n",
       "1  6026eaf8b402c68ae42b1112                 this needs to be investigated   \n",
       "\n",
       "                      event             insert_time       tag  \n",
       "0  6026eaf7b402c68ae42b1110 2021-02-12 20:54:15.970  maneuver  \n",
       "1  6026e9a4b402c68ae42b110b 2021-02-12 20:54:16.131   unknown  "
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "orex.get_annotations()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
